Head mounted information processing apparatus and head mounted display system

ABSTRACT

In a head mounted information processing apparatus, a control unit includes a virtual object generation processing unit configured to generate a virtual object to be displayed by the display unit. The virtual object generation processing unit is configured to generate a first virtual object that is arranged so as to be associated with a first real space where a user exists and a second virtual object that is arranged so as to be associated with a second real space different from the first real space. The control unit is configured to cause the display to display the first virtual object and the second virtual object in accordance with virtual object display instruction information for instructing display of the first virtual object and the second virtual object, which is inputted through an operational input interface.

TECHNICAL FIELD

The present invention relates to a head mounted information processingapparatus and head mounted display system, and more particularly, thepresent invention relates to a technique effective for grasping aposition in a virtual object.

BACKGROUND ART

In recent years, a virtual reality (Virtual Reality: VR) technique, anaugmented reality (Augmented Reality: AR) technique, or a mixed reality(Mixed Reality: MR) technique has been used widely.

Virtual reality is a technique that allows a user to create a virtualworld imitating reality and experience a feeling of being there.Augmented reality is a technique that gives digital information to areal world, and reflects and expands a virtual space (and virtualobjects) created by CG (Computer Graphics) or the like in the realspace. Mixed reality is a technique that combines and fuses a virtualworld artificially created by CG or the like with information in a realworld.

As a tool that embodies these techniques, a head mounted informationprocessing apparatus that is mounted on a head of a user and includes adisplay and a camera is used widely. In the head mounted informationprocessing apparatus, in order to increase the reality of a virtualobject, a display method is put into practical use in which the virtualobject is associated with a spatial coordinate of a real space to appearas if a real object is present there.

In such a display method, a target virtual object can be recognizedvisually by going to a real space where the target virtual object isassociated and arranged, and this allows an intuitive operation system.However, there has been a problem that the target virtual object cannotbe browsed or operated unless a user goes to the real space with whichthe target virtual object is associated.

As a technique for solving this problem, there is a technique in whichby causing at least a part of an augmented reality object in a realworld space, a wearer can easily access the augmented reality objecteven though the wearer moves around the real world space (see Patentdocument 1, for example).

RELATED ART DOCUMENTS Patent Documents

-   Patent document 1: Japanese Patent Application Publication No.    2018-505472

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

As the technique of Patent document 1 described above, Patent document 1describes that the virtual object is displayed so as to be kept in avisual field of a user even though the visual field moves. However, inPatent document 1, no consideration is given to display of a virtualobject that is present in another real space. Therefore, there is aproblem that it is difficult for the user to conveniently view andoperate a virtual object that is present in another real space.

It is an object of the present invention to provide a technique capableof easily grasping an existence position and the like of a virtualobject that is present in another real space different from a realspace.

The foregoing and other objects, features and advantages of the presentinvention will become more readily apparent from the following detaileddescription of embodiments of the present invention that proceeds withreference to the appending drawings.

Means for Solving the Problem

An outline of representative invention of the present inventiondisclosed in the present application will briefly be explained asfollows.

Namely, a representative head mounted information processing apparatusincludes an operational input interface, a camera unit, a display unit,and a control unit. Information is inputted through the operationalinput interface. The camera unit is configured to photograph a realspace. The display unit is configured to display a photographic imagephotographed by the camera unit. The control unit is configured tocontrol the display unit.

Further, the control unit includes a virtual object generationprocessing unit configured to generate a virtual object to be displayedby the display unit. The virtual object generation processing unit isconfigured to generate a first virtual object and a second virtualobject, the first virtual object being arranged so as to be associatedwith a first real space in which a user exists, the second virtualobject being arranged so as to be associated with a second real spacedifferent from the first real space.

Moreover, the control unit is configured to cause the display unit todisplay the first virtual object and the second virtual object inaccordance with virtual object display instruction information forinstructing display of the first virtual object and the second virtualobject, the virtual object display instruction information beinginputted through the operational input interface.

Effects of the Invention

Effects obtained by the representative invention of the presentinvention disclosed in the present application will briefly be explainedas follows.

Since it is possible to visually recognize an arrangement place of avirtual object that is arranged in a different real space accurately, itis possible to improve convenience thereof.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 is a block diagram illustrating one example of a configuration ofa head mounted information processing apparatus according to a firstembodiment;

FIG. 2 is an explanatory drawing illustrating one example of a panoramicview of the surroundings of a usage situation in the head mountedinformation processing apparatus illustrated in FIG. 1;

FIG. 3 is an explanatory drawing illustrating one example of display ofa list of a group of virtual objects by the head mounted informationprocessing apparatus illustrated in FIG. 1;

FIG. 4 is an explanatory drawing illustrating another example of thedisplay of the list of the group of virtual objects illustrated in FIG.3;

FIG. 5 is an explanatory drawing illustrating still another example ofthe display of the list of the group of virtual objects illustrated inFIG. 4;

FIG. 6 is an explanatory drawing illustrating one example of a usagesituation in the head mounted information processing apparatusillustrated in FIG. 1;

FIG. 7 is an explanatory drawing illustrating one example of a displayscreen of the group of virtual objects displayed in a list in theexample of the panoramic view of the surroundings illustrated in FIG. 6;

FIG. 8 is an explanatory drawing illustrating another example of theusage situation illustrated in FIG. 6;

FIG. 9 is an explanatory drawing illustrating one example of a displayscreen of a group of virtual objects displayed in a list in the exampleof the panoramic view of the surroundings illustrated in FIG. 8;

FIG. 10 is an explanatory drawing illustrating one example of display ofthe group of virtual objects by the head mounted information processingapparatus illustrated in FIG. 1;

FIG. 11 is an explanatory drawing illustrating another example of thedisplay of the group of virtual objects illustrated in FIG. 10;

FIG. 12 is an explanatory drawing illustrating one example of switchingdisplay of the group of virtual objects by the head mounted informationprocessing apparatus illustrated in FIG. 1;

FIG. 13 is an explanatory drawing illustrating one example of scalingand a posture operation of the virtual object by the head mountedinformation processing apparatus illustrated in FIG. 1;

FIG. 14 is an explanatory drawing illustrating another example of FIG.13;

FIG. 15 is an explanatory drawing illustrating one example of apanoramic view of the surroundings in a case where all virtual objectsin a plurality of real spaces are viewed;

FIG. 16 is an explanatory drawing illustrating one example of display ofthe virtual objects in a state when viewed from a direction opposite torear entrances illustrated in FIG. 15;

FIG. 17 is an explanatory drawing illustrating another example of thedisplay of the virtual objects illustrated in FIG. 16;

FIG. 18 is an explanatory drawing illustrating still another example ofthe display of the list of the group of virtual objects illustrated inFIG. 16;

FIG. 19 is an explanatory drawing illustrating one example of amulti-display screen by the head mounted information processingapparatus illustrated in FIG. 1;

FIG. 20 is an explanatory drawing illustrating another display exampleof FIG. 19; and

FIG. 21 is a block diagram illustrating one example of a configurationof a head mounted display system according to a second embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

In all drawings for explaining embodiments, the same reference numeralsare respectively in principle applied to the same members, and repeatedexplanation thereof will be omitted.

First Embodiment

Hereinafter, an embodiment will be described in detail.

<Configuration Example of Head Mounted Information Processing Apparatus>

FIG. 1 is a block diagram illustrating one example of a configuration ofa head mounted information processing apparatus according to a firstembodiment.

As illustrated in FIG. 1, a head mounted information processingapparatus 100 includes a camera unit 111, a right eye visual linedetecting unit 112, a left eye visual line detecting unit 113, avibration generating unit 117, an outer peripheral sound microphone 118,a vocalized sound microphone 119, a headphone 120, an operational inputinterface 121, a display 122, a control unit 125, a memory 124, a depthsensor 142, an acceleration sensor 143, a gyro sensor 144, a geomagneticsensor 145, and a stimulus generating unit 146. These functional blocksare connected to each other via a bus 140.

The camera unit 111 is configured to photograph a landscape in front ofa user. The display 122, which is a display unit, displays aphotographic image of a real space, which is photographed by the cameraunit 111. The camera unit 111 may be composed of a plurality of cameras,or may be a 360° entire celestial sphere camera capable of photographingan entire celestial sphere image by combining one or a plurality ofcameras.

The control unit 125 is configured to execute a program 126 (will bedescribed later) stored in the memory 124 to control the respectivefunctional blocks, thereby controlling an operation of the whole headmounted information processing apparatus 100.

The control unit 125 includes a display control unit 151, a datamanaging unit 152, a video processing unit 153, a virtual object postureoperation processing unit 154, a virtual object generation processingunit 155, and a virtual object deformation operation processing unit156.

The virtual object generation processing unit 155 is configured togenerate a group of virtual objects composed of at least one virtualobject that is present in another virtual space different from the realspace. Further, the virtual object generation processing unit 155 isconfigured to arrange the generated group of virtual objects so as to beassociated with the real space.

Here, a group of virtual objects arranged so as to be associated with afirst real space, which is visually recognized or displayed on thedisplay 122, is a group of first virtual objects. Further, a group ofvirtual objects arranged so as to be associated with a second realspace, which is a real space different from the first real space, is agroup of second virtual objects.

Similarly, a group of virtual objects arranged so as to be associatedwith a third real space, which is a real space different from the firstand second real spaces, is a group of third virtual objects. Further, agroup of virtual objects arranged so as to be associated with a fourthreal space, which is a real space different from the first to third realspaces, is a group of fourth virtual objects.

The virtual object generation processing unit 155 is configured togenerate a virtual object on the basis of model data on a virtual objectread out from the memory 124 in response to a user operation inputtedfrom the operational input interface 121.

Note that the model data on the virtual object are not always necessary,and virtual object data may be directly generated without any model by auser operation. For example, in order to generate a rectangularparallelepiped virtual object, eight points to become respectivevertices of a virtual object are specified in the real space by a useroperation using the operational input interface 121.

The virtual object posture operation processing unit 154 is configuredto rotate, enlarge, and reduce the virtual object displayed on thedisplay 122 to operate so as to have posture having a shape that isvisually recognized easily. This is called a posture operation. a resultof the posture operation is not reflected to the posture, the shape, theorientation, and the like of the original virtual object.

The virtual object deformation operation processing unit 156 isconfigured to execute a deformation operation for the virtual objectdisplayed on the display 122. For example, the deformation operation isan operation of changing the orientation of the virtual object, changinga size thereof, changing the shape thereof, deleting a part thereof, ordeleting the whole thereof. A result of the deformation operation by thevirtual object deformation operation processing unit 156 is alsoreflected to the posture, the shape, the orientation, and the like ofthe original virtual object.

The video processing unit 153 is configured to processes video dataphotographed by the camera unit 111, and stores the processed video datain the memory 124 as information data 127. The video processing unit 153is configured to simplify the video data photographed by the camera unit111 mainly for the purpose of reducing the amount of display data andimproving visibility thereof. For example, in a case where there is asquare bookshelf in video data, the video data are simplified bysimplifying a shape of the square bookshelf as a shape such as arectangular parallelepiped with an outer shape having the samedimensions.

Further, the video processing unit 153 is configured to execute videoprocessing for facilitating recognition of each space. For example, thevideo processing unit 153 generates video data from a bird's eye view soas to look down at a space where the user exists at the time ofphotographing on the basis of the video data photographed by the cameraunit 111 from a line of sight of the user.

The display control unit 151 is configured to appropriately combine theinformation data 127 stored in the memory 124 to generate data fordisplay, and cause the display 122 to display it. In this case, theinformation data 127 are the virtual object data, the video dataphotographed by the camera unit 111, and data for display afterprocessing, which are generated by the video processing unit 153.

The data managing unit 152 is configured to manage photographic videodata photographed by the camera unit 111, the data on the virtualobject, and the data for display after processing, which are generatedby the video processing unit 153.

The control unit 125 is composed of a CPU (Central Processing Unit)including a processor dedicated to each arithmetic processing, such as aGPU (Graphics Processing Unit), and the like. The control unit 125 isconfigured to execute the program 126 memorized and stored in the memory124 to control each functional block, thereby controlling an operationof the whole head mounted information processing apparatus 100. Theprogram 126 is a program such as an OS (Operating System) of the headmounted information processing apparatus 100 and an application forcontrolling an operation.

The control unit 125 is configured to control the display control unit151 to arrange and display a group of virtual objects arranged so as tobe associated with the other real space than the first real space, forexample, the group of second virtual objects arranged so as to beassociated with the second real space in the first real space inaccordance with virtual object display instruction information inputtedfrom the operational input interface 121. This makes it possible tovisually recognize and operate a group of virtual objects arranged so asto be associated with the other real space than the first real space.

In this regard, when the group of virtual objects generated by thevirtual object generation processing unit 155 is displayed in a displayvisual field of the display 122, for example, an entire celestial sphereimage indicating the entire surrounding landscape from the head mountedinformation processing apparatus 100 is projected and reflected on thedisplay 122, and a group of virtual objects is arranged at predeterminedpositions of the entire celestial sphere image thus reflected.

In this case, the control unit 125 controls the display control unit 151to arrange the group of first virtual objects and a group of virtualobjects arranged so as to be associated with another real space otherthan the first real space, for example, the group of second virtualobjects arranged so as to be associated with the second real space inthe first real space, and display them.

Further, the control unit 125 may in turn switch between the group offirst virtual objects and the group of second virtual objects arrangedso as to be associated with the second real space, and display them on adisplay screen of the display 122.

Moreover, the control unit 125 may arrange and display the group offirst virtual objects and a group of virtual objects arranged so as tobe associated with another real space other than the first real space ona display screen of the display from visual field positions in the firstreal space.

Further, the control unit 125 may reduce the respective display screensillustrating the groups of virtual objects arranged so as to beassociated with the corresponding real space to display multiple displayscreens side by side; return the display screen of the group of virtualobjects in the selected real space to a normal size to display thedisplay screen; and allow the user to visually recognize and operate adesired virtual object arranged in the selected real space.

Alternatively, the control unit 125 may control scaling (enlargement orreduction) or an operation of posture of a virtual object that isarranged in the display visual field and is hardly recognized visuallyso that the entire shape of the virtual object can be visuallyrecognized by the virtual object posture operation processing unit, anddisplay the virtual object after the posture operation by using thedisplay control unit 151.

The memory 124 is a nonvolatile memory exemplified as a flash memory,and stores various kinds of the program 126 and the information data 127to be used by the control unit 125. The information data 127 are datasuch as data on the groups of virtual objects, coordinate positioninformation of the groups of virtual objects, and photographic images.

The display 122 is composed of a liquid crystal panel or the like, andis configured to display virtual objects, a photographic image of a realspace, and the like. Further, the display 122 is configured to display,on the display screen, the display content such as presentationnotification information to the user and an operation state.

For example, when the photographic image photographed by the camera unit111 or the virtual object is displayed, the photographic image or thevirtual object is arranged and displayed at a predetermined position onthe entire celestial sphere image that indicates the entire surroundinglandscape of the head mounted information processing apparatus 100.Further, the display 122 is configured to multi-display groups ofvirtual objects respectively associated with a plurality of real spacesin the display screen.

The right eye visual line detecting unit 112 is configured to detect avisual line of a right eye of the user. The left eye visual linedetecting unit 113 is configured to detect a visual line of a left eyeof the user. Note that a well-known technique generally used as an eyetracking process may be used for a process of detecting a visual line.

For example, in a method using corneal reflection, there is known atechnique in which a face is irradiated with an infrared ray LED (LightEmitting Diode) and is photographed by an infrared ray camera, aposition (corneal reflection) of the reflected light generated by theinfrared ray LED irradiation on a cornea is used as a reference point,and a visual line is detected on the basis of a position of a pupil withrespect to a position of the corneal reflection.

The acceleration sensor 143 is a sensor for detecting acceleration thatis a change in velocity per unit time, by which motion, vibration, andimpact can be captured. The gyro sensor 144 is a sensor for detectingangular velocity in a rotation direction, by which a state of vertical,horizontal, and diagonal postures can be captured. Therefore, by usingthe acceleration sensor 143 and the gyro sensor 144, it is possible todetect motion of the head of the user equipped with the head mountedinformation processing apparatus 100.

The geomagnetic sensor 145 is a sensor for detecting magnetic force ofthe earth, by which a direction to which a body of the head mountedinformation processing apparatus 100 is facing can be detected. By usinga 3-axis type that detects geomagnetism in a vertical direction inaddition to those in a front-back direction and a horizontal directionas the geomagnetic sensor 145 to capture a change in the geomagnetismwith respect to the motion of the head, it is also possible to detectthe motion of the head.

These sensors allow to detect in detail motion and variation of the headmounted information processing apparatus 100 worn by the user.

The depth sensor 142 is configured to measure a distance to an object ina plane. For example, as the depth sensor 142, there is one in whichreflection of an infrared ray or laser is used. However, it may berealized by the other method such as a method of obtaining distanceinformation from parallax of images respectively photographed by aplurality of cameras whose installation positions are different fromeach other.

The control unit 125 can detect motion of a hand and motion of a body byanalyzing the distance information obtained by the depth sensor 142. Inorder to analyze the motion of the hand and the motion of the body,information obtained from the image photographed by the camera unit 111may be used together.

The stimulus generating unit 146 is configured to generate a stimulusthat can be perceived by skin under the control of the control unit 125.The stimulus generating unit 146 is configured to convert notificationinformation transmitted by the head mounted information processingapparatus 100 to the user into the stimulus that can be perceived by theskin.

The stimulus that can be perceived by the skin includes pressure,thermal sensing, cold sensing, and an electrical stimulus. The stimulusgenerating unit 146 can surely convey notification to the user bygenerating the stimulus that can be perceived by the skin on the head ofthe user to which the head mounted information processing apparatus 100is closely worn.

The vibration generating unit 117 is a unit that generates vibrationunder the control of the control unit 125, and is composed of avibrator, haptics, force feedback, or the like, for example. Thevibration generating unit 117 is configured to convert the notificationinformation to the user into vibration. The vibration generating unit117 can surely convey the notification to the user by generating thevibration on the head of the user to which the head mounted informationprocessing apparatus 100 is closely worn.

The outer peripheral sound microphone 118 and the vocalized soundmicrophone 119 are configured to respectively collect voice from theoutside and user's own vocalization. The vocalized sound microphone 119may be a voice input device such as a bone conduction microphone.

The headphone 120 is to be worn on ears of the user, and is configuredto cause the user to listen to voice to the user. The headphone 120 cannotify the user of the notification information by the voice. Theheadphone 120 may be a voice output device such as a speaker or a boneconduction earphone.

The operational input interface 121 is composed of a keyboard, keybuttons, or a touch pad, for example, and is configured to set and inputinformation that the user wants to input. The operational inputinterface 121 may be provided at a position where the user can carry outan input operation easily.

The operational input interface 121 may be configured to be separatedfrom the body of the head mounted information processing apparatus 100and be connected thereto by wire or wirelessly. As examples of anoperational input device to be separated from the head mountedinformation processing apparatus 100, there are a space mouse, acontroller device, and the like.

The space mouse is a three-dimensional space position input device usinga gyro sensor, an acceleration sensor, or the like. The controllerdevice is configured to detect a spatial position of the controllerdevice itself worn on a body of the user from camera video photographingthe body or information from various kinds of sensors embedded in thecontroller device, and input the spatial position.

The operational input interface 121 may cause the display 122 to displayan input operation screen on the display screen thereof, and captureinput operation information on the basis of a position on the inputoperation screen to which the visual line detected by the right eyevisual line detecting unit 112 and the left eye visual line detectingunit 113 is facing.

The operational input interface 121 may capture the input operationinformation by displaying a pointer on the input operation screen andcausing the user to operate the pointer via the operational inputinterface 121. Further, the user may output voice indicating an inputoperation, and the operational input interface 121 may collect it by thevocalized sound microphone 119 to capture input operation information.

By using the vocalization or the display as the input operation in thismanner, it is possible to further improve usability of the head mountedinformation processing apparatus worn on the head.

The configuration described above allows the head mounted informationprocessing apparatus 100 to display the group of virtual objectsarranged so as to be associated with another real space other than thefirst real space, for example, the group of second virtual objectsarranged so as to be associated with the second real space in the firstreal space so as to overlap with the group of first virtual objects inaccordance with a virtual object display request instruction, inputtedvia the operational input interface 121, for instructing display of thevirtual object, or display the group of first virtual objects or thegroup of second virtual objects so as to be switched therebetween.

Further, by displaying all groups of virtual objects including the groupof virtual objects, which is present in another real space, in thevisual field positions of the first real space, it is possible tofacilitate browse and an operation of the virtual object that is presentin another real space. Moreover, in a case where the number of groups ofvirtual objects is large, it is possible to solve a problem that theuser does not know in which real space the target virtual object ispresent.

Further, it becomes possible to operate the virtual object arranged inthe other real space without moving from the real space that the user isviewing. For example, in a case where a user confirms or writes his orher schedule in another real space B in a state where a calendar isarranged on a wall of a real space A as a virtual object, the user canbrowse and operate the calendar as the virtual object without moving tothe real space A.

This example is a deformation operation against the calendar that is thevirtual object, and is processed by the virtual object deformationoperation processing unit 156. A result of the deformation operation bythe virtual object deformation operation processing unit 156 isreflected to the original object. Therefore, for example, when aschedule Z is written from the real space B to the calendar on the wallof the real space A as the deformation operation, the written schedule Zcan also be recognized visually even though the user views the actualcalendar on the wall of the real space A.

<Operation Example of Head Mounted Information Processing Apparatus>

Subsequently, an operation of the head mounted information processingapparatus 100 will be described.

FIG. 2 is an explanatory drawing illustrating one example of a panoramicview of the surroundings of a usage situation in the head mountedinformation processing apparatus 100 illustrated in FIG. 1. FIG. 3 is anexplanatory drawing illustrating one example of display of a list of agroup of virtual objects by the head mounted information processingapparatus 100 illustrated in FIG. 1. FIG. 3 illustrates a displayexample when the group of virtual objects display as a list is arrangedso as to fall within the display screen of the display 122 in thepanoramic view of the surroundings of the usage situation illustrated inFIG. 2.

In FIG. 2, a user 200 who wears the head mounted information processingapparatus 100 is positioned at the center of a first room 201, and is ina state of viewing a direction 203 opposite to a rear entrance door 202.

A desk 204 and a personal computer 205 are placed on a front side of theuser 200, and a bookshelf 206 is placed on a back side of the user 200.Virtual objects 211 to 213 are a group of first virtual objects, and aregenerated by the virtual object generation processing unit 155.

The virtual object 211 is arranged in front of the user 200. The virtualobject 212 is arranged on a right side of the desk 204. The virtualobject 213 is arranged on a right side of the bookshelf 206 behind theuser 200.

In panoramic view of the surroundings state of the first room 201illustrated in FIG. 2, the user 200 visually recognizes a first realspace projected in the direction 203 of the first room 201 by visuallyobserving the first real space directly or displaying a real imagephotographed by the camera unit 111 on the display 122.

As illustrated in FIG. 3, with respect to visual recognition of thegroup of first virtual objects, the virtual objects 211 to 213 aredisplayed in a list. In this list display, the entire celestial sphereimage photographed by the camera unit 111 is projected and reflected onthe display screen of the display 122, and all the virtual objects aredisplayed on the entire celestial sphere image thus projected andreflected. At this time, the virtual objects 211 to 213 are respectivelyarranged at predetermined positions.

The display control unit 151 causes the display 122 to display thevirtual objects 211 to 213 on the basis of data read out by the datamanaging unit 152. The data managing unit 152 reads out shape data andarrange coordinate data of the virtual objects 211 to 213, which arestored in the memory 124 as the information data 127, and outputs themto the display control unit 151. At that time, the virtual objectposture operation processing unit 154 executes a posture operation forthe virtual objects as needed.

This makes it possible for the user to visually recognize all thevirtual objects that exist in the panoramic view of the surroundingstogether with existence positions thereof.

Note that portions illustrated by dotted lines in FIG. 3 arephotographic objects, thereby intelligibly illustrating a positionalrelationship between the photographic objects and the virtual objects.Therefore, it is not necessary to display the photographic objectillustrated by this dotted line.

FIG. 4 is an explanatory drawing illustrating another example of displayof a list of the group of virtual objects illustrated in FIG. 3. FIG. 4illustrates an example in which the photographic objects illustrated bythe dotted line is not displayed and only the virtual objects 211 to 213are displayed.

FIG. 5 is an explanatory drawing illustrating still another example ofthe display of the list of the group of virtual objects illustrated inFIG. 4. When all the virtual objects existing in the panoramic view ofthe surroundings are to be displayed, as illustrated in FIG. 5, thephotographic objects including the desk 204, the personal computer 205,and the bookshelf 206, which are illustrated by the dotted lines in FIG.3, may be displayed as background images. This makes it possible to moreeasily recognize the positional relationship between the desk 204, thepersonal computer 205, and the bookshelf 206 in the real space and thevirtual objects 211 to 213.

For example, as illustrated in FIG. 2, in a case where the virtualobject 213 is provided on an upper right side of the bookshelf 206 onthe back side, it is possible to easily recognize the positions of thevirtual objects so long as the bookshelf 206 is displayed on a lowerleft side of the virtual object 213 as a background image.

As described above, the display control unit 151 generates the data fordisplaying the virtual objects and the like, and causes the display 122to display the virtual objects. The display control unit 151 reads, fromthe memory 124, data on shapes and display positions of the objectscontained in the information data 127, background video photographed bythe camera unit 111, data generated by the video processing unit 153,and the like, and generates display data from the read data.

Further, the virtual object posture operation processing unit 154executes the posture operation for the virtual objects as needed, andadjusts a display position, a size, and a shape thereof so that each ofthe virtual objects is to be displayed at a corresponding position onthe screen. This adjustment is executed by the virtual object postureoperation processing unit 154 on the basis of commands of the program126 stored in the memory 124.

Note that the entire celestial sphere image that indicates the entiresurrounding landscape may be obtained by using the entire solid angleand the entire celestial sphere camera that can photograph an image ofthe entire celestial sphere at one time. Alternatively, a plurality ofimages photographed by a camera with a normal angle of view may bejoined by the video processing unit 153 to generate the entire celestialsphere image.

Further, an image of only a part of a range that can be obtained may beused for display of a landscape image. For example, it is a case whereexistence of almost all the virtual objects can be visually recognizednot by the entire celestial sphere but by a part of a range of an imagesuch as a hemispherical image of the upper half of the entire solidangle.

FIG. 6 is an explanatory drawing illustrating one example of a usagesituation in the head mounted information processing apparatus 100illustrated in FIG. 1. FIG. 7 is an explanatory drawing illustrating oneexample of a display screen of the group of virtual objects displayed ina list in the example of the panoramic view of the surroundingsillustrated in FIG. 6.

FIG. 6 illustrates a state where the user 200 who wears the head mountedinformation processing apparatus 100 is positioned in the vicinity of anentrance door 402 of a second room 401 and views an indoor direction 403of the second room 401. Further, a television stand 404 and a television405 are installed on a front side of the user 200. A shelf 406 isinstalled on a wall on a right side when viewed from the user 200.

The group of second virtual objects generated by the virtual objectgeneration processing unit 155 is composed of virtual objects 411 to413. In FIG. 7, the virtual object 411 is positioned on a rear upperside of the television 405. The virtual object 412 is positioned on arear right side of the television 405. The virtual object 413 ispositioned in the vicinity of the wall on a left side of the entrancedoor 402.

In the panoramic view of the surroundings of the second room 401illustrated in FIG. 6, the user 200 directly observes the second realspace projected in the indoor direction 403 visually, or visuallyrecognizes an image of the real space photographed by the camera unit111 and displayed on the display 122.

With respect to the visual recognition of the group of second virtualobjects, as illustrated in FIG. 7, an entire celestial sphere imageindicating the entire surrounding landscape is projected and reflectedon the display screen of the display 122, and the respective virtualobjects 411 to 413 are arranged at predetermined positions in the entirecelestial sphere image thus projected and reflected to be displayed in alist.

This makes it possible for the user to visually recognize existence ofall the virtual objects existing in the panoramic view of thesurroundings together with existence positions thereof in the similarmanner to the case illustrated in FIG. 3. Further, when all the virtualobjects existing in the whole panoramic view of the surroundings are tobe displayed, the photographic objects such as the television stand 404,the television 405, and the shelf 406, which are illustrated in FIG. 7by the dotted lines, may be displayed as background images.

This makes it possible to easily recognize the positional relationshipbetween the real spaces and the virtual objects. In this regard, sincethe process of displaying them on the display 122 is similar to thatillustrated in FIG. 3, explanation thereof will be omitted.

FIG. 8 is an explanatory drawing illustrating another example of theusage situation illustrated in FIG. 6. FIG. 9 is an explanatory drawingillustrating one example of a display screen of the group of virtualobjects displayed in a list in the example of the panoramic view of thesurroundings illustrated in FIG. 8. FIG. 8 illustrates a state where theuser 200 who wears the head mounted information processing apparatus 100is positioned in the center of a third room 601 and is viewing a leftlateral direction 603 behind an entrance door 602. A board 604 ispositioned in front of the user 200, and a window 605 is positioned on aright side of the user 200. Further, a clock 606 is arranged on a rightside of the window 605.

The group of third virtual objects generated by the virtual objectgeneration processing unit 155 is composed of virtual objects 611 to613. In FIG. 8, the virtual object 611 is positioned on a left side ofthe board 604, and the virtual object 612 is positioned above the window605. The virtual object 613 is positioned behind the user 200.

In the panoramic view of the surroundings state in the third room 601illustrated in FIG. 8, the user 200 directly observes the third realspace projected in the direction 603 visually, or visually recognizesthe image of the real space photographed by the camera unit 111 anddisplayed on the display 122.

With respect to the visual recognition of the group of third virtualobjects, as illustrated in FIG. 9, the entire celestial sphere imageindicating the entire surrounding landscape is projected and reflectedon the display screen of the display 122, and the respective virtualobjects 611 to 613 are arranged at positions where they exist in theentire celestial sphere image thus projected and reflected to bedisplayed in a list.

This makes it possible for the user to visually recognize existence ofall the virtual objects existing in the panoramic view of thesurroundings together with existence positions thereof in the similarmanner to those illustrated in FIG. 3 and FIG. 7. Further, when all thevirtual objects existing in the whole panoramic view of the surroundingsare to be displayed, the photographic objects such as the board 604, thewindow 605, and the clock 606, which are illustrated in FIG. 9 by thedotted lines, may be displayed as background images.

This makes it possible to easily recognize the positional relationshipbetween the real spaces and the virtual objects. In this regard, sincethe process of displaying them on the display 122 is similar to thatillustrated in FIG. 3, explanation thereof will be omitted.

FIG. 10 is an explanatory drawing illustrating one example of display ofthe group of virtual objects by the head mounted information processingapparatus illustrated in FIG. 1. In the examples illustrated in FIG. 3,FIG. 7, and FIG. 9, the groups of virtual objects respectively arrangedso as to be associated with the real spaces are visually recognized inthe respective real spaces. However, in these examples, the group ofvirtual objects arranged so as to be associated with another real spaceother than the real space that is being projected currently cannot bevisually recognized in the real space currently projected.

FIG. 10 describes a display example in which the respective groups ofvirtual objects arranged so as to be associated with the real spaces arearranged on the display screen of the display 122 so as to overlap witheach other while keeping the visual field positions of the real spaces.

Since the portions illustrated in FIG. 3, FIG. 7, and FIG. 9 to whichthe same reference numerals are respectively assigned in FIG. 10 havethe similar operations as the operations that have already beendescribed in FIG. 3, FIG. 7, and FIG. 9, detailed explanation thereofwill be omitted.

FIG. 10 illustrates a state where the group of second virtual objectsarranged so as to be associated with the second real space and the groupof third virtual objects arranged so as to be associated with the thirdreal space are displayed in the first real space currently projected onthe display screen of the display 122 so as to overlap with each otherin addition to the group of first virtual objects arranged so as to beassociated with the first real space.

The group of first virtual objects is composed of the virtual objects211, 212, and 213. The group of second virtual objects is composed ofthe virtual objects 411, 412, and 413. The group of third virtualobjects is composed of the virtual objects 611, 612, and 613.

As illustrated in FIG. 10, by arranging and displaying the virtualobjects 411 to 413, and 611 to 613 arranged in the other real spaces inthe real space currently present so as to overlap with each other, it ispossible to visually recognize all the virtual objects arranged so as tobe associated with the other real spaces without switching the realspaces.

This makes it possible to visually recognize a desired virtual objecteasily from the display screen on which all virtual objects aredisplayed even though there are a large number of real spaces and groupsof virtual objects. Moreover, it becomes possible to easily carry out adesired operation such as modification of the selected virtual object.As a result, it is possible to improve usability.

The above operation is to be displayed on the display 122 by the displaycontrol unit 151. Further, the virtual object posture operationprocessing unit 154 executes a posture operation for any virtual objectas needed.

The virtual object posture operation processing unit 154 adjusts adisplay position, a size, a shape, and the like so that a virtual objectis displayed at a corresponding position on the screen in accordancewith an instruction of the program 126 stored in the memory 124. Thedisplay control unit 151 generates display data from the data adjustedby the virtual object posture operation processing unit 154 to displayit on the display 122.

FIG. 11 is an explanatory drawing illustrating another example of thedisplay of the group of virtual objects illustrated in FIG. 10. FIG. 11illustrates an example in which virtual objects displayed on the displayscreen of the display 122 are arranged at substantially the samecoordinate positions and the virtual objects are thereby displayed so asto overlap with each other.

Since the portions illustrated in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG.8, FIG. 9, and FIG. 10 to which the same reference numerals arerespectively assigned in FIG. 11 have the similar operations to theoperations that have already been described in FIG. 2, FIG. 3, FIG. 6,FIG. 7, FIG. 8, FIG. 9, and FIG. 10, detailed explanation thereof willbe omitted.

The virtual objects 212 and 412 illustrated in FIG. 10 are virtualobjects that are arranged and overlap with each other at substantiallythe same coordinate position, and the overlap reduces visibility.Therefore, as illustrated in FIG. 11, the virtual objects 212 and 412 tobe displayed so as to overlap with each other are displayed by shiftingvirtual objects 901 and 902 to coordinate positions where they do notoverlap with each other.

Further, a mark 903 is displayed at the original coordinate positionwhere the virtual objects 212 and 412 are arrange, and virtual lines 904and 905 are respectively displayed so as to connect to each otherbetween the virtual objects 901 and 902 and the mark 903 that arearranged and displayed by shifting the coordinate positions.

Thus, by shifting the virtual objects arranged and displayed so as tooverlap with each other at substantially the same position anddisplaying them in this manner, it is possible to display virtualobjects without overlapping with each other. Further, by displaying themark 903, it is possible to easily recognize the original position wherethe virtual object is arranged. Moreover, by displaying the virtuallines 904 and 905, it is possible to further improve visibility of thevirtual objects.

Further, instead of shifting the display position of the virtual object,the virtual objects that are arranged and displayed at the same positionso as to overlap with each other may be displayed as a translucentimage, that is, a transparent image.

The above operation is to be displayed on the display 122 by the displaycontrol unit 151. Further, the virtual object posture operationprocessing unit 154 executes a posture operation for any virtual objectas needed.

The posture operation by the virtual object posture operation processingunit 154 is that a display position, a size, and a shape are adjusted onthe basis of the program 126 stored in the memory 124 so that thevirtual object is displayed at the corresponding position.

Hereinafter, unless otherwise specified, the display of the virtualobject on the display 122 is subjected to the similar processing.

Further, the display of the group of virtual objects may be that theuser selects a desired virtual object from the virtual objects 212 and412 (FIG. 10) displayed so as to be superimposed with each other, andonly the group of virtual objects similarly associated therewith isdisplayed in the real space arranged so as to be associated with theselected virtual object. In that case, the group of virtual objectsarranged so as to be associated with the other virtual space is notdisplayed.

This makes it possible to easily select a next desired virtual objectfrom the group of virtual objects arranged so as to be associated withthe same real space as that of the desired virtual object.

<Switching Display Example of Group of Virtual Objects>

FIG. 12 is an explanatory drawing illustrating one example of switchingdisplay of the group of virtual objects by the head mounted informationprocessing apparatus 100 illustrated in FIG. 1. Since the portionsillustrated in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG. 8, FIG. 9, and FIG.10 to which the same reference numerals are respectively assigned inFIG. 12 have the similar operations as the operations that have alreadybeen described in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG. 8, FIG. 9, andFIG. 10, detailed explanation thereof will be omitted.

As illustrated in FIG. 11, FIG. 12 illustrates an example in which allthe virtual objects are displayed so as not to overlap with each other,but groups of virtual objects arranged so as to be associated withrespective real spaces are displayed so as to be switched in turn.

A display screen 1001 is a display screen on which a group of firstvirtual objects arranged so as to be associated with a first real spaceis first displayed. A display screen 1002 is a display screen on which agroup of second virtual objects arranged so as to be associated with asecond real space is displayed.

A display screen 1003 is a display screen on which a group of thirdvirtual objects arranged so as to be associated with a third real spaceis displayed. A display screen 1004 is a display screen on which a groupof virtual objects arranged so as to be associated with another realspace different from the first to third real spaces is displayed in acase where there is such a group of virtual objects. These displayscreens 1001 to 1004 are displayed on the display 122 so as to beswitched in turn.

This makes it possible to visually recognize only the group of virtualobjects arranged so as to be associated with each of the real spaces inturn instead of visually recognizing all the groups of virtual objectsat once. As a result, it is possible to visually recognize a desiredvirtual object from each of the groups of virtual objects efficiently,and it becomes possible to further improve visibility.

Further, in the switching display of the groups of virtual objects, theymay be switched and displayed at regular intervals when they can beeasily recognized visually by means of an input operation such asswiping from the operational input interface 121, for example. Thismakes it possible to further heighten the visibility of the virtualobjects.

In a case where the user wants to take more time to visually recognizethe virtual objects in detail or the user wants to switch to a nextscreen in a short time in the switching display of the virtual objectsat regular intervals, increase or decrease of the time to visuallyrecognize the group of virtual objects being viewed may be changed bymeans of an operation input of the operational input interface 121.

When each of the display screens 1001, 1002, and 1003 is displayed, notonly the group of virtual objects but also the photographic image of thereal space corresponding to the group of virtual objects may bedisplayed as a background.

In this case, by displaying the background of the photographic image, itbecomes possible to easily recognize the virtual objects. Note that thephotographic background may be photographed by the camera unit 111illustrated in FIG. 1 are stored in the memory 124. In a case where anangle of view of the camera unit 111 is narrow, images photographedseparately may be brought together and used.

<Scaling and Posture Operation of Virtual Object>

Subsequently, an operation of the scaling and posture operation for thevirtual objects by the head mounted information processing apparatus 100will be described. FIG. 13 is an explanatory drawing illustrating oneexample of the scaling and posture operation for the virtual objects bythe head mounted information processing apparatus illustrated in FIG. 1.FIG. 14 is an explanatory drawing illustrating another example of FIG.13.

Since the portions illustrated in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG.8, FIG. 9, and FIG. 10 to which the same reference numerals arerespectively assigned in FIG. 13 and FIG. 14 have the similar operationsas the operations that have already been described in FIG. 2, FIG. 3,FIG. 6, FIG. 7, FIG. 8, FIG. 9, and FIG. 10, detailed explanationthereof will be omitted.

FIG. 13 illustrates an example in which the virtual object 612 that istoo small to be viewed is selected from the virtual objects 211 to 213,411 to 413, 611 to 613 displayed in a list to heighten visibility of theselected virtual object 612.

In this case, the virtual object posture operation processing unit 154executes an operation to enlarge a shape of the virtual object 612;moves and arranges the enlarged virtual object toward a predeterminedposition in front thereof; and displays it as a virtual object 1101. Atthat time, the list display of all the virtual objects 211 to 213, 411to 413, and 611 to 613 is left in a background portion in the displayscreen.

Alternatively, the virtual object posture operation processing unit 154may execute an enlargement operation after moving the virtual object 612toward the front without leaving it. The predetermined position is setto a position where it is easy to visually recognize the virtual object612 as an initial value. This initial value is stored in advance in thememory 124, for example.

Further, the initial value may be set by the control unit 125 writingsetting information inputted by the user through the operational inputinterface 121 into the memory 124. For example, by setting a movablerange portion of a hand in front of the body as the initial value thatis the position defined in advance, it is easy to visually recognize thevirtual object, and it is also easy to carry out the posture operationand the deformation operation.

The operation to move the virtual object toward the position defined inadvance may be moved and arranged automatically by selecting anarbitrary virtual object using the operational input interface 121 bythe user. Alternatively, it may be arranged mutually by a naturaloperation such as pulling the selected object. Further, when it isarranged, the user may carry out an operation to determine anenlargement ratio using the operational input interface 121 to changethe enlargement ratio.

The above operation is that the control unit 125 controls the virtualobject posture operation processing unit 154 and the like in accordancewith a user operation inputted from the operational input interface 121.The virtual object posture operation processing unit 154 changesinformation on a shape and a display position of the selected object.

The display control unit 151 reads the information data 127 on the shapeand the display position of the virtual object stored in the memory 124,and displays it on the display 122.

This makes it possible to visually recognize the virtual object that istoo small to be viewed in the list display more clearly. The selectedvirtual object is returned to the original position before anarrangement operation of the background portion in the display screen bycontrol of the control unit 125 after visually confirming. Thisoperation is executed automatically after an operation to end visualconfirmation.

Subsequently, in a case where an operation to select another virtualobject to move and arrange it to the front, the previously selectedvirtual object returns to the original position thereof, and a listdisplay image of the original virtual objects remains as a background.This makes it possible to easily carry out the arrangement operation andvisual confirmation of a next virtual object.

It is possible to visually recognize existence of all the groups ofvirtual objects in the display in which the virtual objects are arrangedin the entire celestial sphere image. However, it may be difficult tovisually recognize the entire shape of the virtual objects.

Therefore, with respect to the virtual object arranged on the displayscreen of the display 122 and that it is difficult to visuallyrecognize, posture of the virtual object is operated by the virtualobject posture operation processing unit 154 so that it is easy tovisually recognize the entire shape of the virtual object to display thevirtual object.

For example, the virtual object 411 illustrated in FIG. 14 originallyhas a cubic shape, but becomes a display shape that cannot be visuallyrecognized as a cube by a display shape of the entire celestial sphereimage. The virtual object posture operation processing unit 154 firstmoves the virtual object 411 toward a display position in front of whichit is easy to operate posture thereof while enlarging it.

Then, the virtual object posture operation processing unit 154 rotatesthe virtual object 411 after movement including a three-dimensionalrotational operation, and executes a posture operation to a displayshape by which it is easy to visually recognize the entire shape,thereby being converted into a display shape indicated by a virtualobject 1201 and displayed.

The virtual object 1201 may be returned to the original position beforethe arrangement operation as the virtual object 411 after the virtualobject 1201 is visually confirmed.

Thus, the virtual object posture operation processing unit 154 executesthe posture operation for the virtual object whose entire shape isdifficult to be recognized visually to convert it into a virtual objectwith a display shape by which the entire shape can be visuallyrecognized, whereby it becomes possible to visually recognize and graspthe entire shape and the whole aspect of the virtual object accurately.

The operation to display the virtual object in the display shape bywhich it is easy to visually recognize the entire shape may be displayedby a display shape stored in advance in the memory 124 without carryingout the posture operation by the user, for example. For the displayshape whose entire shape is easy to visually recognize, information onorientation, a size, and a color by which the object is visuallyrecognized easily may be stored in advance in the shape data that becomea model at the time of generating the virtual object in the memory 124as posture information, and the posture information may be used bytaking over it to the generated virtual object. Further, the user mayspecify the posture information for each virtual object, thereby storingthe posture information in the memory 124, and using it at the time ofdisplay thereof.

<Deformation Operation of Virtual Object>

Moreover, the virtual object deformation operation processing unit 156can execute the deformation operation for the virtual object. Thevirtual object deformation operation processing unit 156 reads the shapeand the display position of the virtual object, which are stored in thememory 124; changes information on the shape and the display position ofthe selected virtual object; and writes the changed information into thememory 124. The shape of the virtual object contains orientation, asize, an angle, and the like.

The display control unit 151 reads information written into the memory124, and displays the virtual object subjected to the deformationoperation on the display 122 on the basis of the information.

A result of the deformation operation by the virtual object deformationoperation processing unit 156 is also reflected to the display state ofthe original virtual object. In a case where the orientation of thevirtual object is changed by the deformation operation, the orientationof the virtual object itself is changed. Therefore, the virtual objectis displayed so as to overlap with the landscape of the real space. Forexample, even in normal display states illustrated in FIG. 2, FIG. 6,and FIG. 8, the virtual object is displayed in the orientation after thedeformation operation.

At the time of the deformation operation, by displaying the shape of theoriginal virtual object before the deformation, which containsorientation, a size, and the like, by semi-transparent or displaying itat a place in the visual field, which is not used in the deformationoperation, a difference between the original virtual object before thedeformation and the shape of the virtual object after the deformation,which contains orientation, a size, and the like, may be displayed in aneasily understood manner.

Whether any of the posture operation and the deformation operation is tobe carried out for the virtual object is specified before the operationby an operation mode switching button (not illustrated in the drawings)or the like provided in the head mounted information processingapparatus 100, for example.

By combining the posture operation and the deformation operation, thedeformation operation may be carried out after the virtual object isenlarged and easily viewed by the posture operation. Further, anoperation applied to the virtual object by the posture operation, suchas rotation, enlargement, reduction, or the like may be applied to thedeformation operation.

<Display Example of Group of Virtual Objects of Another Real Space>

Subsequently, an operation when all groups of virtual objects arrangedso as to be associated with another real space from the present realspace are displayed will be described with reference to FIG. 15 to FIG.18.

Since the portions illustrated in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG.8, FIG. 9, and FIG. 10 to which the same reference numerals arerespectively assigned in FIG. 15 to FIG. 18 have the similar operationsas the operations that have already been described in FIG. 2, FIG. 3,FIG. 6, FIG. 7, FIG. 8, FIG. 9, and FIG. 10, detailed explanationthereof will be omitted.

FIG. 15 is an explanatory drawing illustrating one example of apanoramic view of the surroundings in a case where all virtual objectsin a plurality of real spaces are viewed.

In FIG. 15, as well as the case illustrated in FIG. 2, a user 200 whowears a head mounted information processing apparatus 100 is positionedat the center of a first room 201, and is in a state where the user 200views a direction 203 opposite to an entrance door 202.

Further, a desk 204, a personal computer 205, and a bookshelf 206 areplaced in the first room 201. As well as FIG. 2, virtual objects 211,212, and 213 are arranged in the first room 201 as a group of firstvirtual objects generated by a virtual object generation processing unit155.

A second room 401 is arranged to the left of the first room 201, and athird room 601 is arranged to the right of the first room 201. Atelevision stand 404, a television 405, and a shelf 406 are placed inthe second room 401.

As well as FIG. 6, virtual objects 411, 412, and 413 are arranged in thesecond room 401 as a group of second virtual objects generated by thevirtual object generation processing unit 155.

Aboard 604, a window 605, and a clock 606 are arranged in the third room601. As well as FIG. 8, virtual objects 611, 612, and 613 are arrangedin the third room 601 as a group of third virtual objects generated bythe virtual object generation processing unit 155.

FIG. 16 is an explanatory drawing illustrating one example of display ofthe virtual objects in a state when viewed from a direction 203 oppositeto the entrance door 202 at the rear of FIG. 15. As illustrated in FIG.16, the virtual objects 411 to 413 and 611 to 613 arranged so as to beassociated with another real space are displayed in a display screen bymaking objects that partition the real spaces, such as a room wall,transparent.

This makes it possible to visually recognize all the virtual objects 211to 213, 411 to 413, and 611 to 613 easily regardless of the real spaces,and it is possible to improve usability when selecting a desired virtualobject.

Each of the virtual objects is displayed so that the display positionthereof allows the real space with which the corresponding virtualobject is arranged so as to be associated to be recognized easily.

For example, in FIG. 16, the virtual objects 611 to 613 are positionedand displayed on the right side of the display screen. This makes itpossible to easily recognize that the group of virtual objects isarranged so as to be associated with the third room 601 illustrated inFIG. 15.

Similarly, the virtual objects 411 to 413 are positioned and displayedon the left side of the display screen. This makes it possible to easilyrecognize that the group of virtual objects is arranged so as to beassociated with the second room 401 illustrated in FIG. 15.

FIG. 17 is an explanatory drawing illustrating another example of thedisplay of the virtual objects illustrated in FIG. 16. Here, in thedisplay example illustrated in FIG. 16, there is a fear that if anotherroom, that is, another real space is far away, a group of virtualobjects arranged so as to be associated with the real space is displayedso as to be small, whereby it is difficult for the user to visuallyrecognize the group of virtual objects.

In such a case, the virtual objects 411 to 413 are enlarged anddisplayed as virtual objects 1514 to 1516. Similarly, the virtualobjects 611 to 613 are enlarged and displayed as virtual objects 1511 to1513.

The virtual object posture operation processing unit 154 executes anoperation to enlarge a shape for each of the virtual objects 411 to 413,and 611 to 613 to generate the virtual objects 1511 to 1516 thusenlarged.

This makes it possible to visually recognize the virtual objects, whichare too small to be viewed, more clearly. Note that at the time of theoperation to enlarge the shape of the virtual object by the virtualobject posture operation processing unit 154, the shape of the virtualobject may be enlarged to a size that is easily handled and viewed.

Further, with respect to the virtual object 1511, the virtual object 411illustrated in FIG. 16 is displayed in a display shape in which theentire shape thereof is visually recognized easily on the basis of theposture information described above and specified in advance. Byspecification of the user, a part of the virtual objects or all of thevirtual objects may be displayed in the display shape in which theentire shape thereof is visually recognized easily.

As illustrated in FIG. 16, by making the room walls or the like thatblock another real space from the present real space transparent anddisplaying all the virtual objects, it is possible to easily identifythe real space with which the selected virtual object is arranged so asto be associated.

FIG. 18 is an explanatory drawing illustrating still another example ofthe display of the list of the group of virtual objects illustrated inFIG. 16. FIG. 18 illustrates a display example when the displayillustrated in FIG. 10 is shifted to the display illustrated in FIG. 16.Virtual objects 611 a, 612 a, and 613 a correspond to the virtualobjects illustrated in FIG. 10. Virtual objects 611 b, 612 b, and 613 bare virtual objects when the display illustrated in FIG. 10 is shiftedto the display illustrated in FIG. 16.

Namely, when the display illustrated in FIG. 10 is shifted to thedisplay illustrated in FIG. 16, as illustrated in FIG. 18, the virtualobjects 611 a, 612 a, and 613 a move as the virtual objects 611 b, 612b, and 613 b so as to gather together to a right side portion of thedisplay screen.

This makes it possible to easily recognize that these virtual objects611 b, 612 b, and 613 b are virtual objects arranged so as to beassociated with the third real space of the third room 601, which ispositioned on the right side of the screen of the first room 201.

At the time of this display shift, by slowly and gradually shifting thedisplay at a speed where the user can follow the virtual objects withhis or her eyes, it is possible for the user to visually recognize whichreal space the group of moving virtual objects is the virtual objectsarranged so as to be associated with surely.

Although the example in which an arbitrary group of virtual objects ismoved has been described in FIG. 18, all the virtual objects may bemoved at the same time. Further, only selected one or a plurality ofselected virtual objects may be moved and displayed, and the othervirtual objects that are not selected may not be displayed.

Subsequently, a case where a group of virtual objects arranged so as tobe associated with each real space is multi-displayed will be described.

FIG. 19 is an explanatory drawing illustrating one example of amulti-display screen by the head mounted information processingapparatus illustrated in FIG. 1. FIG. 19 illustrates an example in whichthe display screens illustrated in FIG. 3, FIG. 7, and FIG. 9 arereduced and multi-displayed on the display screen of the display 122.

Since the portions illustrated in FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG.8, FIG. 9, and FIG. 10 to which the same reference numerals arerespectively assigned in FIG. 17 have the similar operations as theoperations that have already been described in FIG. 2, FIG. 3, FIG. 6,FIG. 7, FIG. 8, FIG. 9, and FIG. 10, detailed explanation thereof willbe omitted.

In FIG. 19, a display screen 1701, a display screen 1702, a displayscreen 1703, and a display screen 1704 are displayed on the displayscreen of the display 122 in order from the upper left.

The group of first virtual objects arranged so as to be associated withthe first real space is displayed on the display screen 1701. The groupof second virtual objects arranged so as to be associated with thesecond real space is displayed on the display screen 1702. The group ofthird virtual objects arranged so as to be associated with the thirdreal space is displayed on the display screen 1703. The group of fourthvirtual objects arranged so as to be associated with the fourth realspace is displayed on the display screen 1704.

In the display screen 1704 on which the group of fourth virtual objectsis displayed, virtual objects 1705 and 1706 are arranged so as to beassociated with the fourth real space composed of a landscape includingbuildings, a vehicle, humans, and the like.

In a case where a desired virtual object is selected, the user searchesthe display screens 1701 to 1704 obtained by reducing the group ofvirtual objects arranged so as to be associated with each of the realspaces illustrated in FIG. 19 and multi-displaying them side by side.Then, the user changes the display screen on which the selected virtualobject exists into a normal full display screen to become a state wherethe desired virtual object can be visually recognized easily.

This makes it possible to easily select the real space, in which thedesired virtual object is arranged, from the multi-display screen onwhich the groups of virtual objects respectively arranged so as to beassociated with the real spaces are displayed. Moreover, since only thegroup of virtual objects arranged so as to be associated with theselected virtual object is displayed, it becomes possible to visuallyrecognize the desired virtual object in an easily visible state.

Note that FIG. 19 illustrates the 4-multi display in which the fourdisplay screens are displayed, but a multi-display method may havevarious variations such as 9-multi display or 16-multi display, forexample.

Further, as illustrated in FIG. 20, a plurality of spaces may bedisplayed side by side in a bird's eye view. In this case, the videoprocessing unit 153 generates an image having a bird's eye view, and thedisplay control unit 151 displays the image generated by the videoprocessing unit 153 on the display 122. This makes it easier to imagethe space, and it is possible to facilitate selection of the space inwhich the desired virtual object is included.

When a bird's-eye video of the space is generated, by displaying anavatar of the user himself or herself at a position where the user iscurrently present in the video, it is possible to easily grasp the spacewhere the user himself or herself exists.

Further, in a case where any other user exists in the plurality ofdisplayed spaces, coordinate information on the space where the otheruser exists may be obtained by wireless communication or the like, andthe other user may be displayed at the corresponding coordinate positionon the bird's-eye space.

Similarly, existence of the user himself or herself or the other usermay be displayed on the image of the multi-display by an avatar or amark. This makes it possible to easily grasp a positional relationshipbetween the spaces and each user.

The position of the user himself or herself on the space can bespecified by using various kinds of sensor information, for example, adistance to the wall obtained by the depth sensor, an image photographedby the camera unit 111, and the like. An image of the avatar to bedisplayed is stored in advance in the memory 124 as the information data127. Then, the video processing unit 153 synthesizes video of the spaceusing the information data 127 stored in the memory 124.

The coordinate information of the other user is obtained from variouskinds of sensors or a camera of an information terminal worn by theuser, and is transmitted from a communication interface included in theinformation terminal. The head mounted information processing apparatus100 directly receives the coordinate information through a communicationinterface 1804. Alternatively, the head mounted information processingapparatus 100 may receive the coordinate information via a server (notillustrated in the drawings).

The state where the virtual objects are arranged in the real space wherethe user exists has been described as the examples with reference toFIG. 2 to FIG. 19. However, the similar operations can be made even in acase where the space itself handled by the user is a virtual space andvirtual objects are arranged in the virtual space, such as VR.

Further, the virtual object can be used as a reminder. The virtualobject generation processing unit 155 can generate virtual objects in aspace other than the space where the user currently exists.

By using this function, the virtual object can be used as the reminder.For example, the virtual object generation processing unit 155 generatesa virtual object of umbrella and arranges it at an entrance while theuser is staying in a living room, thereby becoming a reminder toremember the umbrella when the user goes out.

Note that information indicating the request instruction by the userthrough the operational input interface 121, the operation by the headmounted information processing apparatus 100, and a display operationmay be displayed on the display 122.

Alternatively, the information described above may be notified to theuser by being vocalized to the user by voice from the headphone 120,generating vibration by the vibration generating unit 117 that is inclose contact with the user, or generating stimulus by the stimulusgenerating unit 146.

As a result, it is possible to cause the user to notify and recognize anoperating state of the head mounted information processing apparatus 100surely.

Further, the input operations for the operation executed by the headmounted information processing apparatus 100 and the operation of thedisplay may be taken not only from the input operation by theoperational input interface 121, but also from an input operation basedon a motion obtained by detecting the motion of the hand of the user bythe camera unit 111 or the like, for example.

As described above, the virtual objects can easily be browsed andoperated even though the virtual objects exist in another real space.

Second Embodiment

<Configuration Example of Head Mounted Display System>

FIG. 21 is a block diagram illustrating one example of a configurationof a head mounted display system 1801 according to a second embodiment.

As illustrated in FIG. 21, the head mounted display system 1801 includesa head mounted information processing apparatus 100 and a virtual objectgenerating server apparatus 1802. Each of the head mounted informationprocessing apparatus 100 and the virtual object generating serverapparatus 1802 is connected to a network 1803.

The head mounted information processing apparatus 100 illustrated inFIG. 21 is newly provided with a communication interface 1804 and atransmission/reception antenna 1805 in addition to the configuration ofthe respective functional blocks to which the same reference numeralsillustrated in FIG. 1 are assigned. On the other hand, the head mountedinformation processing apparatus 100 illustrated in FIG. 21 is notprovided with a virtual object generation processing unit 155.

The virtual object generating server apparatus 1802 includes a virtualobject generation processing unit 1811, a memory 1812, a control unit1813, a communication interface 1814, a transmission/reception antenna1815, and the like. The respective functional blocks in the virtualobject generating server apparatus 1802 are connected to each other viaa bus 1820. Note that in FIG. 21, the same reference numerals arerespectively assigned to the same processing units as the firstembodiment illustrated in FIG. 1, and explanation thereof will beomitted.

In the head mounted display system 1801, the virtual object generationprocessing unit 1811 included in the virtual object generating serverapparatus 1802 is configured to generate virtual objects.

The virtual objects generated by the virtual object generationprocessing unit 1811 are stored in the memory 1812. The communicationinterface 1814 is configured to transmit the virtual objects stored inthe memory 1812 to the head mounted information processing apparatus 100via the network 1803 that is a communication network from thetransmission/reception antenna 1815. The head mounted informationprocessing apparatus 100 receives the virtual objects transmitted viathe network 1803.

Further, in FIG. 21, a process of displaying the virtual objects by thehead mounted information processing apparatus 100 itself is similar tothat in the first embodiment. However, the second embodiment isdifferent from the first embodiment in that the virtual objectgenerating server apparatus 1802 that is an apparatus other than thehead mounted information processing apparatus 100 generates virtualobjects.

In the virtual object generating server apparatus 1802, the memory 1812is a nonvolatile semiconductor memory such as a flash memory in thesimilar manner to the memory 124 of the head mounted informationprocessing apparatus 100.

Various kinds of programs and the generated virtual objects to be usedby the control unit 1813 of the virtual object generating serverapparatus 1802 are stored in the memory 1812. The communicationinterface 1814 is a communication interface that communicates with thehead mounted information processing apparatus 100 via the network 1803,and is configured to transmit and receive information to and from thehead mounted information processing apparatus 100.

The control unit 1813 is composed of a CPU or the like, for example. Thecontrol unit 1813 is configured to execute an OS or programs such as anapplication for controlling an operation, which are memorized and storedin the memory 1812, thereby controlling each of the functional blocks,and controlling the whole virtual object generating server apparatus1802.

The control unit 1813 is configured to control generation of the virtualobject by the virtual object generation processing unit 1811, storage ofthe generated virtual objects in the memory 1812, and the like. Further,the control unit 1813 controls the communication interface 1814 totransmit the generated virtual objects to the head mounted informationprocessing apparatus 100 in accordance with a request to transmit andoutput the virtual objects from the head mounted information processingapparatus 100.

This makes it possible to generate the virtual objects not by the headmounted information processing apparatus 100, but by the virtual objectgenerating server apparatus 1802 separated from the head mountedinformation processing apparatus 100.

As a result, it is possible to increase a scale of the amount of virtualobject information that can be handled. It becomes possible to generatevirtual objects respectively required by a plurality of head mountedinformation processing apparatuses 100 at a plurality of places at thesame time and distribute them thereto.

As described above, it is possible to easily visually recognize andoperate the virtual objects arranged in separate real spaces by theplurality of head mounted information processing apparatuses 100 at thesame time.

As described above, the invention made by inventors of the presentapplication has been described specifically on the basis of theembodiments. However, the present invention is not limited to theembodiments described above, and it goes without saying that the presentinvention may be modified into various forms without departing from thesubstance thereof.

Note that the present invention is not limited to the embodimentsdescribed above, and various modifications are contained. For example,the embodiments described above have been explained in detail forexplaining the present invention clearly. The present invention is notnecessarily limited to one that includes all configurations that havebeen explained.

Further, a part of the configuration of one embodiment can be replacedby a configuration of the other embodiment. Further, a configuration ofthe other embodiment can be added to a configuration of one embodiment.Further, a part of the configuration of each of the embodiments can beadded to the other configuration, deleted, or replaced thereby.

Further, a part or all of the respective configuration described above,the functions, processing units, and processing functions may berealized by hardware that is designed by an integrated circuit, forexample. Further, the respective configuration described above and thefunctions may be realized by software so that a processor interpretsprograms realizing the respective functions and execute the interpretedprograms. Information on programs, tables, and files, which realize therespective functions, can be placed in a recording device such as amemory, a hard disk, or an SSD (Solid State Drive), or a recordingmedium such as an IC card, an SD card, or a DVD.

Further, control lines and information lines are illustrated so long asthey are thought to be necessary for explanation. All of the controllines and the information line are not necessarily illustrated on aproduct. In fact, it may be considered that almost all of the componentsare connected to each other.

REFERENCE SIGNS LIST

-   100 head mounted information processing apparatus-   111 camera unit-   112 right eye visual line detecting unit-   113 left eye visual line detecting unit-   117 vibration generating unit-   118 outer peripheral sound microphone-   119 vocalized sound microphone-   120 headphone-   121 operational input interface-   122 display-   124 memory-   125 control unit-   140 bus-   142 depth sensor-   143 acceleration sensor-   144 gyro sensor-   145 geomagnetic sensor-   146 stimulus generating unit-   151 display control unit-   152 data managing unit-   153 video processing unit-   154 virtual object posture operation processing unit-   155 virtual object generation processing unit-   156 virtual object deformation operation processing unit-   1801 head mounted display system-   1802 virtual object generating server apparatus-   1803 network-   1804 communication interface-   1805 transmission/reception antenna-   1811 virtual object generation processing unit-   1812 memory-   1813 control unit-   1814 communication interface-   1815 transmission/reception antenna

1. A head mounted information processing apparatus, comprising: anoperational input interface through which information is inputted; acamera unit configured to photograph a real space; a display unitconfigured to display a photographic image photographed by the cameraunit; and a control unit configured to control the display unit, whereinthe control unit includes a virtual object generation processing unitconfigured to generate a virtual object to be displayed by the displayunit, wherein the virtual object generation processing unit isconfigured to generate a first virtual object and a second virtualobject, the first virtual object being arranged so as to be associatedwith a first real space in which a user exists, the second virtualobject being arranged so as to be associated with a second real spacedifferent from the first real space, and wherein the control unit isconfigured to cause the display unit to display the first virtual objectand the second virtual object in accordance with virtual object displayinstruction information for instructing display of the first virtualobject and the second virtual object, the virtual object displayinstruction information being inputted through the operational inputinterface.
 2. The head mounted information processing apparatusaccording to claim 1, wherein the control unit is configured to causethe display unit to display the second virtual object at a coordinate ofa same position as that in a case where the second virtual is displayedin the second real space.
 3. The head mounted information processingapparatus according to claim 2, wherein the control unit is configuredto cause the display unit to display the first virtual object or thesecond virtual object as a transparent image in a case where the firstvirtual object and the second virtual object are to be displayed so asto overlap with each other.
 4. The head mounted information processingapparatus according to claim 2, wherein the control unit is configuredto cause the display unit to display the first virtual object and thesecond virtual object so as to be shifted in a case where the firstvirtual object and the second virtual object are to be displayed so asto overlap with each other.
 5. The head mounted information processingapparatus according to claim 4, wherein the control unit is configuredto generate a display mark and virtual lines and cause the display unitto display the first virtual object and the second virtual objecttogether with the display mark and the virtual lines, the display markindicating a display position where the first virtual object and thesecond virtual object overlap with each other, the virtual linesrespectively connecting the display mark to the first virtual object andthe second virtual object.
 6. The head mounted information processingapparatus according to claim 1, wherein the control unit is configuredto cause the display unit to switch between the first virtual object andthe second virtual object and display the switched one in accordancewith the virtual object display instruction information inputted fromthe operational input interface.
 7. The head mounted informationprocessing apparatus according to claim 1, wherein the control unit isconfigured to switch between the first virtual object and the secondvirtual object every fixed time, and cause the display unit to displaythe switched one.
 8. The head mounted information processing apparatusaccording to claim 1, wherein the control unit is configured to causethe display unit to display a photographic image of the first real spacephotographed by the camera unit when the first virtual object and thesecond virtual object are displayed.
 9. The head mounted informationprocessing apparatus according to claim 1, wherein the control unitincludes a virtual object posture operation processing unit configuredto operate posture of the first virtual object or the second virtualobject selected by the operational input interface, and wherein thevirtual object posture operation processing unit is configured tooperate the posture of the first virtual object or the second virtualobject selected by the virtual object posture operation processing unit.10. The head mounted information processing apparatus according to claim1, wherein the control unit includes a virtual object deformationoperation processing unit configured to execute a deformation operationof deforming the first virtual object or the second virtual objectselected by the operational input interface, and wherein the virtualobject deformation operation processing unit is configured to executethe deformation operation for the first virtual object or the secondvirtual object selected by the virtual object deformation operationprocessing unit.
 11. The head mounted information processing apparatusaccording to claim 1, wherein the control unit is configured to generatea first display screen for displaying the first virtual object arrangedso as to be associated with the first real space and a second displayscreen for displaying the second virtual object arranged so as to beassociated with the second real space, and cause the display unit todisplay the generated first display screen and the generated seconddisplay screen side by side.
 12. The head mounted information processingapparatus according to claim 11, wherein the control unit is configuredto enlarge, when any of the first display screen or the second displayscreen is selected by the operational input interface, the selectedscreen, and cause the display unit to display the enlarged screen.
 13. Ahead mounted display system, comprising: a head mounted informationprocessing apparatus connected to a communication network, the headmounted information processing apparatus being configured to display areal space object and a virtual object; and a virtual object generatingserver apparatus connected to the communication network, wherein thehead mounted information processing apparatus includes: an operationalinput interface through which information is inputted; a display unitconfigured to display the virtual object; and a control unit configuredto control the display unit, wherein the virtual object generatingserver apparatus includes: a virtual object generation processing unitconfigured to generate the virtual object; and a communication interfacevia which information is transmitted to and received from thecommunication network, wherein the virtual object generation processingunit is configured to generate a first virtual object and a secondvirtual object in accordance with virtual object display instructioninformation for instructing display of the virtual object inputted fromthe operational input interface, the first virtual object being arrangedso as to be associated with a first real space in which a user exists,the second virtual object being arranged so as to be associated with asecond real space different from the first real space, wherein thecommunication interface is configured to transmit, to the communicationnetwork, the first virtual object and the second virtual objectgenerated by the virtual object generation processing unit, and whereinthe control unit is configured to cause the display unit to display thefirst virtual object and the second virtual object transmitted via thecommunication network.